Moving yandex provider to new provider structure #46525
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kunaljubce
requested review from
potiuk,
ashb and
jedcunningham
as code owners
boring-cyborg
bot
added
area:dev-tools
area:providers
kind:documentation
provider:yandex
labels
86 tasks
potiuk
approved these changes
Feb 6, 2025
|
OK. there is a small caveat with "latest botocore" - Let me add a fixup - I know how to fix it |
|
When I try to reproduce this error locally with =========================== short test summary info ============================
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/ml_engine/example_mlengine.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/ml_engine/example_mlengine.py': 'Traceback (most recent call last):\n File "<frozen importlib._bootstrap_external>", line 1040, in get_data\n File "/opt/airflow/airflow/utils/timeout.py", line 69, in handle_timeout\n raise AirflowTaskTimeout(self.error_message)\nairflow.exceptions.AirflowTaskTimeout: DagBag import timeout for /opt/airflow/providers/google/tests/system/google/cloud/ml_engine/example_mlengine.py after 30.0s.\nPlease take a look at these docs to improve your DAG import time:\n* http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/stable/best-practices.html#top-level-python-code\n* http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/stable/best-practices.html#reducing-dag-complexity, PID: 75\n'}
assert 1 == 0
+ where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/ml_engine/example_mlengine.py': 'Traceback (most recent call...-website.eu-central-1.amazonaws.com/docs/apache-airflow/stable/best-practices.html#reducing-dag-complexity, PID: 75\n'})
+ where {'/opt/airflow/providers/google/tests/system/google/cloud/ml_engine/example_mlengine.py': 'Traceback (most recent call...-website.eu-central-1.amazonaws.com/docs/apache-airflow/stable/best-practices.html#reducing-dag-complexity, PID: 75\n'} = <airflow.models.dagbag.DagBag object at 0xffff48087910>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_forecasting_training.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_forecasting_training.py': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n from google.cloud.aiplatform_v1 import gapic_version as package_version\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
+ where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_forecasting_training.py'...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
+ where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_forecasting_training.py'...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff43775190>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_image_object_detection.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_image_object_detection.py': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n from google.cloud.aiplatform_v1 import gapic_version as package_version\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
+ where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_image_object_detection.p...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
+ where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_image_object_detection.p...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff434f20a0>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_image_training.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_image_training.py': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n from google.cloud.aiplatform_v1 import gapic_version as package_version\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
+ where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_image_training.py': 'Tra...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
+ where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_image_training.py': 'Tra...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff43838430>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_list_training.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_list_training.py': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n from google.cloud.aiplatform_v1 import gapic_version as package_version\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
+ where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_list_training.py': 'Trac...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
+ where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_list_training.py': 'Trac...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff4341ac40>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_tabular_training.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_tabular_training.py': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n from google.cloud.aiplatform_v1 import gapic_version as package_version\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
+ where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_tabular_training.py': 'T...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
+ where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_tabular_training.py': 'T...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff434cab50>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_video_tracking.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_video_tracking.py': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n from google.cloud.aiplatform_v1 import gapic_version as package_version\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
+ where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_video_tracking.py': 'Tra...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
+ where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_video_tracking.py': 'Tra...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff4383ec40>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_video_training.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_video_training.py': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n from google.cloud.aiplatform_v1 import gapic_version as package_version\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
+ where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_video_training.py': 'Tra...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
+ where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_auto_ml_video_training.py': 'Tra...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff43755f40>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_batch_prediction_job.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_batch_prediction_job.py': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n from google.cloud.aiplatform_v1 import gapic_version as package_version\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
+ where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_batch_prediction_job.py': 'Trace...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
+ where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_batch_prediction_job.py': 'Trace...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff43880880>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_container.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_container.py': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n from google.cloud.aiplatform_v1 import gapic_version as package_version\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
+ where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_container.py': 'Traceback...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
+ where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_container.py': 'Traceback...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff4389ae20>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_job.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_job.py': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n from google.cloud.aiplatform_v1 import gapic_version as package_version\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
+ where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_job.py': 'Traceback (most...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
+ where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_job.py': 'Traceback (most...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff43549b20>.import_errors
FAILED tests/always/test_example_dags.py::test_should_be_importable[providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_job_python_package.py] - AssertionError: import_errors={'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_job_python_package.py': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py", line 35, in <module>\n from google.cloud.aiplatform_v1 import gapic_version as package_version\n File "/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/__init__.py", line 57, in <module>\n from .services.index_service import IndexServiceClient\nImportError: cannot import name \'IndexServiceClient\' from partially initialized module \'google.cloud.aiplatform_v1.services.index_service\' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'}
assert 1 == 0
+ where 1 = len({'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_job_python_package.py': '...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'})
+ where {'/opt/airflow/providers/google/tests/system/google/cloud/vertex_ai/example_vertex_ai_custom_job_python_package.py': '...ular import) (/usr/local/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/index_service/__init__.py)\n'} = <airflow.models.dagbag.DagBag object at 0xffff43955cd0>.import_errors
= 12 failed, 1549 passed, 1 skipped, 1 xfailed, 1 warning in 1577.55s (0:26:17) =
No stopped containers
There were errors when running some tasks. Quitting. |
|
pushed likely fix |
|
Thanks @potiuk. For my understanding, can you please explain a little about the botocore issue? |
Yeah. The thing is that with botocore tests we are upgrading to latest (i.e. newest) possible botocore - basically by removing aiobotocore that holds it back. The See: https://pypi.org/project/botocore/#history botocore - released today, aiobotocore released 22 Jan and it has So those tests are removing aiobotocore and try to run all the tests with latest botocore. And yandex also uses a aiobotocore I think. I see my fix did not work, so I have to fix it differently |
potiuk
force-pushed
the
provider-moving/yandex
branch
from
6ba927f to
ebdaeca
Compare
|
Ok. I pushed simplified version. I think in this case we do not have to run |
|
🤞 |
jscheffl
reviewed
Feb 6, 2025
AAAAAH... Merged tooo fast... good eye Jens. |
|
Ah no --... yandex docs were copied previously (accidentally) as part of another provider (TERADATA) :) . All good :) |
insomnes
pushed a commit
to insomnes/airflow
that referenced
this pull request
Feb 6, 2025
* Moving yandex provider to new provider structure * fixup! Moving yandex provider to new provider structure --------- Co-authored-by: Jarek Potiuk <[email protected]>
insomnes
pushed a commit
to insomnes/airflow
that referenced
this pull request
Feb 6, 2025
* Moving yandex provider to new provider structure * fixup! Moving yandex provider to new provider structure --------- Co-authored-by: Jarek Potiuk <[email protected]>
Did not know but good you confirmed.... and no Aaaah :-D |
niklasr22
pushed a commit
to niklasr22/airflow
that referenced
this pull request
Feb 8, 2025
* Moving yandex provider to new provider structure * fixup! Moving yandex provider to new provider structure --------- Co-authored-by: Jarek Potiuk <[email protected]>
ambika-garg
pushed a commit
to ambika-garg/airflow
that referenced
this pull request
Feb 17, 2025
* Moving yandex provider to new provider structure * fixup! Moving yandex provider to new provider structure --------- Co-authored-by: Jarek Potiuk <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Relates to #46045, moving
yandexprovider to new folder structure. Also added a check in themove_providers.pyscript to check if a dir exists before trying to move it.^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.